Skip to main content
Open in Colab Open in Kaggle

​
Installation

First, let’s install the necessary packages to run this tutorial. 
pip install fastdup datasets
Now, test the installation. If there’s no error message, we are ready to go. 
import fastdup
fastdup.__version__

'1.41'
📘 Info There are over 50,000 openly available datasets on Hugging Face Datasets. Some datasets are gated and can only be downloaded if you have a user credential.

​
Load Dataset

The Hugging Face datasets package provides an easy interface to load any datasets from the Hugging Face platform. On top of the package, fastdup provides a wrapper class FastdupHFDataset as a connector to ensure the datasets package works seamlessly within fastdup. The FastdupHFDataset class works the same way as the load_dataset method. You can import the wrapper class and specify the name of the Hugging Face Datasets repository as the first argument. In this example, we load the Tiny ImageNet dataset which contains 100,000 images of 200 classes (500 for each class) downsized to 64×64 colored images. Each class has 500 training images, 50 validation images, and 50 test images. In the following code, we load the train split of the Tiny ImageNet dataset. 
from fastdup.datasets import FastdupHFDataset
dataset = FastdupHFDataset("zh-plus/tiny-imagenet")
👍 Tip Optional parameters for the FastdupHFDataset class:
  • split - Which split to download. Default: 'train'.
  • img_key- The key value for the dataset column containing images. Default: 'image'.
  • label_key - The key value for the dataset column containing labels. Default: 'label'.
  • cache_dir - Where to cache the downloaded dataset. Default: '/root/.cache/huggingface/datasets/'
  • jpg_save_dir - Which folder to store the jpg images. Default: 'jpg_images'
  • reconvert_jpg- Flag to force reconversion of images from .parquet to .jpg. Default: False
See implementation for details.
Now, let’s inspect the dataset object. 
dataset

Dataset({
    features: ['image', 'label'],
    num_rows: 100000
})
Get the first element of the dataset. 
dataset[0]

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=64x64>,
 'label': 0}
Get the PIL image of the first element. 
dataset[0]['image']
Get the label of the first element. 
dataset[0]['label']

0
📘 Info You can also confirm the image and label of the first element by heading to the dataset page.

​
Run fastdup

Once loaded, we can now analyze the dataset in fastdup by passing in 2 properties of dataset into fastdup:
  • dataset.img_dir - Returns the folder directory where the jpg images are saved.
  • dataset.annotations- Returns aDataFrame of image and class labels.
dataset.img_dir

/root/.cache/huggingface/datasets/tiny-imagenet/jpg_images

dataset.annotations
Let’s run fastdup and pass indataset.img_dir and dataset.annotations as arguments. 
fd = fastdup.create(input_dir=dataset.img_dir)
fd.run(annotations=dataset.annotations)

Warning: fastdup create() without work_dir argument, output is stored in a folder named work_dir in your current working path.
FastDup Software, (C) copyright 2022 Dr. Amir Alush and Dr. Danny Bickson.


                                                                               
   ad88                                          88                            
  d8"                             ,d             88                            
  88                              88             88                            
MM88MMM  ,adPPYYba,  ,adPPYba,  MM88MMM  ,adPPYb,88  88       88  8b,dPPYba,   
  88     ""     `Y8  I8[    ""    88    a8"    `Y88  88       88  88P'    "8a  
  88     ,adPPPPP88   `"Y8ba,     88    8b       88  88       88  88       d8  
  88     88,    ,88  aa    ]8I    88,   "8a,   ,d88  "8a,   ,a88  88b,   ,a8"  
  88     `"8bbdP"Y8  `"YbbdP"'    "Y888  `"8bbdP"Y8   `"YbbdP'Y8  88`YbbdP"'   
                                                                  88           
                                                                  88           


2023-09-19 21:40:45 [INFO] Going to loop over dir /tmp/tmpjb5y9uvf.csv
2023-09-19 21:40:45 [INFO] Found total 100000 images to run on, 100000 train, 0 test, name list 100000, counter 100000 
2023-09-19 21:44:25 [INFO] Found total 100000 images to run onmated: 0 Minutes
Finished histogram 38.760
Finished bucket sort 38.971
2023-09-19 21:44:45 [INFO] 20456) Finished write_index() NN model
2023-09-19 21:44:45 [INFO] Stored nn model index file work_dir/nnf.index
2023-09-19 21:44:56 [INFO] Total time took 251595 ms
2023-09-19 21:44:56 [INFO] Found a total of 40 fully identical images (d>0.990), which are 0.02 % of total graph edges
2023-09-19 21:44:56 [INFO] Found a total of 0 nearly identical images(d>0.980), which are 0.00 % of total graph edges
2023-09-19 21:44:56 [INFO] Found a total of 11083 above threshold images (d>0.900), which are 5.54 % of total graph edges
2023-09-19 21:44:56 [INFO] Found a total of 10001 outlier images         (d<0.050), which are 5.00 % of total graph edges
2023-09-19 21:44:56 [INFO] Min distance found 0.601 max distance 1.000
2023-09-19 21:44:56 [INFO] Running connected components for ccthreshold 0.960000 
.0
 ########################################################################################

Dataset Analysis Summary: 

    Dataset contains 100000 images
    Valid images are 100.00% (100,000) of the data, invalid are 0.00% (0) of the data
    Similarity:  0.05% (46) belong to 1 similarity clusters (components).
    99.95% (99,954) images do not belong to any similarity cluster.
    Largest cluster has 4 (0.00%) images.
    For a detailed analysis, use `.connected_components()`
(similarity threshold used is 0.9, connected component threshold used is 0.96).

    Outliers: 6.34% (6,344) of images are possible outliers, and fall in the bottom 5.00% of similarity values.
    For a detailed list of outliers, use `.outliers()`.

########################################################################################
Would you like to see awesome visualizations for some of the most popular academic datasets?
Click here to see and learn more: https://app.visual-layer.com/vl-datasets?utm_source=fastdup
########################################################################################
0
Once completed, we can visualize the issues in fastdup galleries.

​
Duplicates

Let’s visualize the duplicates in a gallery. 
fd.vis.duplicates_gallery()
To get a detailed DataFrame on the duplicates/near-duplicate found, use the similaritymethod. 
similarity_df = fd.similarity()  
similarity_df.head()
We can get the number of duplicates/near-duplicates by filtering them on the distance score. A distance of 1.0 is an exact copy, and vice versa. 
near_duplicates = similarity_df[similarity_df["distance"] >= 0.99]
near_duplicates = near_duplicates[["distance","filename_from", "filename_to", "label_from", "label_to"]]
len(near_duplicates)

40

​
Outliers

fd.vis.outliers_gallery()

outliers_df = fd.outliers()
outliers_df.head()

​
Mislabels

fd.vis.similarity_gallery(slice='diff')

​
Wrap Up

That’s it! We’ve just conveniently surfaced many issues with this dataset by running fastdup. By taking care of dataset quality issues, we hope this will help you train better models. Questions about this tutorial? Reach out to us on our Slack channel!

​
VL Profiler - A faster and easier way to diagnose and visualize dataset issues

The team behind fastdup also recently launched VL Profiler, a no-code cloud-based platform that lets you leverage fastdup in the browser. VL Profiler lets you find:
  • Duplicates/near-duplicates.
  • Outliers.
  • Mislabels.
  • Non-useful images.
Here’s a highlight of the issues found in the RVL-CDIP test dataset on the VL Profiler.
👍 Free Usage Use VL Profiler for free to analyze issues on your dataset with up to 1,000,000 images. Get started for free.
Not convinced yet? Interact with a collection of datasets like ImageNet-21K, COCO, and DeepFashion here. No sign-ups needed.